Qianming Blog
ClassNote_CIS5190
Qianming Huang Three
  2025-10-08 19:43:51   2025-10-09 01:47:41    
  1. 1. ML Design Choices
  2. 2. Linear Regression
    1. 2.1. Linear Functions
    2. 2.2. Loss Function
      1. 2.2.1. Mean Squared Error (MSE)
    3. 2.3. Linear Regression Problem
    4. 2.4. Linear Regression With Feature Maps
      1. 2.4.1. Feature Maps
      2. 2.4.2. Algorithm for Non-linear Regression
    5. 2.5. Training/Test Split
    6. 2.6. Bias and Variance: way to assess overfitting and underfitting
      1. 2.6.1. Effective Bias/Variance Under Varying Dataset Sizes
    7. 2.7. Regularization: way to fix underfitting and overfitting
      1. 2.7.1. Intuition on L2 Regularization
    8. 2.8. Cross-Validation for Hyperparameter Tuning / Model Selection
      1. 2.8.1. Hyperparameter Tuning
      2. 2.8.2. Validation Set
        1. 2.8.2.1. Cross Validation
    9. 2.9. Other Performance Metrics
    10. 2.10. Optimizing the MSE Loss
      1. 2.10.1. Closed-Form Solution
      2. 2.10.2. Gradient Descent
        1. 2.10.2.1. Batch Gradient Descent & Stochastic Gradient Descent
      3. 2.10.3. Optimizing Regularized Linear Regression
        1. 2.10.3.1. L2-regularized Linear Regression
    11. 2.11. Feature Preprocessing And Selection
      1. 2.11.1. Feature Standardization
      2. 2.11.2. Feature map
      3. 2.11.3. Feature Selection
      4. 2.11.4. Handling Missing Values
    12. 2.12. L1 Regularization (LASSO) And Automatic Feature Selection
  3. 3. Logistic Regression
    1. 3.1. Linear Regression to Binary Classification
  4. 4. Neural Networks
    1. 4.1. A Framework Uniting Many ML Hypothesis Class
    2. 4.2. Neural Net Building Blocks and Notation
      1. 4.2.1. Basic Structure
    3. 4.3. Forward Propagation
    4. 4.4. Backward Pass
      1. 4.4.1. Loss functions
      2. 4.4.2. Optimizer
      3. 4.4.3. Back Propagation
      4. 4.4.4. Computational Graph
        1. 4.4.4.1. Computing Rules for Each Nodes
    5. 4.5. The Neural Net Toolbox
      1. 4.5.1. Optimization
        1. 4.5.1.1. Mini-Batch SGD
        2. 4.5.1.2. Momentum Gradient Descent
        3. 4.5.1.3. Adaptive Learing Rates
        4. 4.5.1.4. Set the Learning Rate
      2. 4.5.2. Activation Functions
      3. 4.5.3. Managing Weights
        1. 4.5.3.1. Initialization
        2. 4.5.3.2. Batch Normalization
        3. 4.5.3.3. Regularization
        4. 4.5.3.4. Dropout
      4. 4.5.4. Managing Training
      5. 4.5.5. Data Augmentation
      6. 4.5.6. Hyperparameter Optimization
        1. 4.5.6.1. Coarse-to-fine Search
      7. 4.5.7. Cross-validation vs a Single Validation Set
    6. 4.6. Practical Tips for Training Neural Nets
  5. 5. K-Nearest Neighbors
    1. 5.1. The Definition of “Nearest”
    2. 5.2. “Non-parametric” Machine Learning Approches
    3. 5.3. Hyperparameters in KNN
  1. 1. ML Design Choices
  2. 2. Linear Regression
    1. 2.1. Linear Functions
    2. 2.2. Loss Function
      1. 2.2.1. Mean Squared Error (MSE)
    3. 2.3. Linear Regression Problem
    4. 2.4. Linear Regression With Feature Maps
      1. 2.4.1. Feature Maps
      2. 2.4.2. Algorithm for Non-linear Regression
    5. 2.5. Training/Test Split
    6. 2.6. Bias and Variance: way to assess overfitting and underfitting
      1. 2.6.1. Effective Bias/Variance Under Varying Dataset Sizes
    7. 2.7. Regularization: way to fix underfitting and overfitting
      1. 2.7.1. Intuition on L2 Regularization
    8. 2.8. Cross-Validation for Hyperparameter Tuning / Model Selection
      1. 2.8.1. Hyperparameter Tuning
      2. 2.8.2. Validation Set
        1. 2.8.2.1. Cross Validation
    9. 2.9. Other Performance Metrics
    10. 2.10. Optimizing the MSE Loss
      1. 2.10.1. Closed-Form Solution
      2. 2.10.2. Gradient Descent
        1. 2.10.2.1. Batch Gradient Descent & Stochastic Gradient Descent
      3. 2.10.3. Optimizing Regularized Linear Regression
        1. 2.10.3.1. L2-regularized Linear Regression
    11. 2.11. Feature Preprocessing And Selection
      1. 2.11.1. Feature Standardization
      2. 2.11.2. Feature map
      3. 2.11.3. Feature Selection
      4. 2.11.4. Handling Missing Values
    12. 2.12. L1 Regularization (LASSO) And Automatic Feature Selection
  3. 3. Logistic Regression
    1. 3.1. Linear Regression to Binary Classification
  4. 4. Neural Networks
    1. 4.1. A Framework Uniting Many ML Hypothesis Class
    2. 4.2. Neural Net Building Blocks and Notation
      1. 4.2.1. Basic Structure
    3. 4.3. Forward Propagation
    4. 4.4. Backward Pass
      1. 4.4.1. Loss functions
      2. 4.4.2. Optimizer
      3. 4.4.3. Back Propagation
      4. 4.4.4. Computational Graph
        1. 4.4.4.1. Computing Rules for Each Nodes
    5. 4.5. The Neural Net Toolbox
      1. 4.5.1. Optimization
        1. 4.5.1.1. Mini-Batch SGD
        2. 4.5.1.2. Momentum Gradient Descent
        3. 4.5.1.3. Adaptive Learing Rates
        4. 4.5.1.4. Set the Learning Rate
      2. 4.5.2. Activation Functions
      3. 4.5.3. Managing Weights
        1. 4.5.3.1. Initialization
        2. 4.5.3.2. Batch Normalization
        3. 4.5.3.3. Regularization
        4. 4.5.3.4. Dropout
      4. 4.5.4. Managing Training
      5. 4.5.5. Data Augmentation
      6. 4.5.6. Hyperparameter Optimization
        1. 4.5.6.1. Coarse-to-fine Search
      7. 4.5.7. Cross-validation vs a Single Validation Set
    6. 4.6. Practical Tips for Training Neural Nets
  5. 5. K-Nearest Neighbors
    1. 5.1. The Definition of “Nearest”
    2. 5.2. “Non-parametric” Machine Learning Approches
    3. 5.3. Hyperparameters in KNN